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In  early  2014,  the  U.S.  Department  of  Defense  (DoD)  asked 

the  RAND  National  Defense  Research  Institute  to  conduct  an 
independent  assessment  of  sexual  assault,  sexual  harassment, 
and  gender  discrimination  in  the  military.  The  RAND  Military 
Workplace  Study  (RMWS)  included  one  of  the  largest  surveys 
of  its  kind:  Almost  560,000  active-  and  reserve-component 
service  members  were  invited  to  participate,  and  more  than  170,000 
completed  the  survey. 


The  RMWS  took  a  new  approach  to  counting  service  members 
who  experienced  sexual  assault,  sexual  harassment,  and  gender 
discrimination,  providing  DoD  with  unprecedented  detail  on  the 
frequency,  nature,  and  context  of  these  experiences  among  men  and 
women  in  each  branch  of  service.  DoD  requested  this  redesign  to  better 
align  the  survey  with  legal  definitions  of  sexual  assault  and  sexual 
harassment. 


Because  such  a  large  number  of  service  members  participated  in  the 
survey,  the  RMWS  estimates  were  very  precise,  but  precision  alone 
does  not  guarantee  that  the  study  correctly  counted  the  events  it  was 
designed  to  assess.  Therefore,  the  RAND  team  conducted  a  series 
of  methodological  substudies  to  investigate  whether  the  RMWS 
survey  estimates  suffered  from  other  sources  of  error,  including: 
misclassification  of  sexual  assault  and  harassment,  the  exclusion  of 
some  types  of  service  members  from  the  study  sample,  and  differences 
between  survey  respondents  and  nonrespondents  in  their  risk  of  sexual 
assault  and  harassment.  These  investigations  included  both  additional 
statistical  analyses  of  the  previously  published  RMWS  study  data  and 
analyses  of  new  data  collected  in  follow-up  surveys  of  service  members 
and  from  administrative  records. 

These  investigations  did  not  identify  substantial  bias  or  error  in  the 
previously  reported  RMWS  estimates.  However,  the  evidence  suggests 
that  the  RMWS  estimates  were  more  likely  to  slightly  underestimate 
than  to  overestimate  the  true  rate  of  sexual  assault  in  the  military. 

The  exact  magnitude  of  any  bias  appears  to  be  small  and  cannot  be 
estimated  with  precision. 
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Survey  nonresponse  did  not  substantially  affect  the 
estimates  of  sexual  assault,  sexual  harassment,  and  gender 
discrimination.  If  service  members  who  chose  not  to  participate 
in  the  survey  had  different  sexual  assault  or  sexual  harassment 
experiences  from  those  of  the  weighted  sample  of  respondents, 
this  would  result  in  biased  survey  estimates.  To  investigate  how 
nonrespondents  differed  from  respondents,  we  conducted  follow-up 
surveys  of  three  samples  of  service  members  who  did  not  participate 
in  the  original  RMWS  survey:  (1)  8,060  randomly  selected  members 
who  were  recruited  by  phone  for  a  phone  interview;  (2)  8,083 
randomly  selected  members  who  were  recruited  by  mail  for  a  self- 
administered  paper  survey;  and  (3)  252,741  who  did  not  respond 
in  time  to  be  included  in  the  estimates  but  were  given  additional 
time  to  complete  the  original  RMWS  survey  online.  These  follow¬ 
up  surveys  did  not  identify  a  consistent  pattern  of  bias  in  the 
estimated  prevalence  of  sexual  assault,  sexual  harassment,  or  gender 
discrimination.  Specifically,  both  the  phone  and  web  follow-up  surveys 
revealed  possible  nonresponse  bias  in  the  RMWS  estimates,  but  in 
different  directions:  The  phone  survey  suggested  that  the  RMWS 
overestimated  the  prevalence  of  nonrespondents’  experiences,  while 
the  web-based  follow-up  suggested  underestimation  by  approximately 
the  same  magnitude.  The  follow-up  mailed  survey  was  stronger 
methodologically  but  did  not  identify  evidence  of  significant 
nonresponse  bias  for  sexual  assault  or  harassment.  It  did  suggest, 
however,  that  the  RMWS  estimates  may  have  slightly  undercounted  the 
rate  of  gender  discrimination. 

RMWS  nonresponse  weights  performed  well  in  reducing 
the  nonresponse  bias  in  the  estimated  prevalence 
of  sexual  assault,  sexual  harassment,  and  gender 
discrimination.  Service  members  with  characteristics  associated 
with  a  higher  risk  of  sexual  assault  had  the  lowest  likelihood  of 
responding  to  the  survey.  To  address  this  potential  source  of  bias,  we 
developed  new  and  powerful  nonresponse  weighting  methods  that  built 
on  those  used  in  the  earlier  DoD  surveys  and  by  the  broader  survey 
research  community.  In  a  series  of  statistical  analyses,  we  found  that 
the  new  weights  eliminated  more  sources  of  potential  bias  from  the 
RMWS  estimates  than  the  weighting  procedures  used  in  the  previous 
DoD  studies. 

Survey  estimates  compared  well  to  official  records. 

The  RMWS  estimate  of  2,435  official  reports  of  sexual  assault  to 
DoD  in  the  year  prior  to  the  survey  slightly  underestimated  the  true 
number  of  reports  filed  (2,997).  This  discrepancy  could  result  from 
uncertainties  in  whether  official  reports  were  made  by  those  survey 
respondents  who  indicated  they  were  “not  sure”  if  they  had  made  such 
a  report,  or  because  some  service  members  who  filed  official  reports 
may  not  have  been  eligible  to  take  the  survey  because,  for  instance, 
they  separated  from  the  military  during  the  past  year.  Nevertheless,  we 
regard  this  as  further  weak  evidence  that  the  RMWS  is  more  likely  to 
have  slightly  underestimated  the  true  prevalence  of  sexual  assault. 


Undercounting  and  overcounting 
sexual  assault 

The  RMWS  sampling  plan,  survey  questionnaire,  and 
analysis  plan  all  required  judgments  that  affected  how  we 
counted  incidents  and  the  prevalence  of  sexual  assault. 

We  examined  the  effects  of  the  following  decisions 
on  the  RMWS  estimate  that  20,300  service  members 
experienced  a  sexual  assault  in  the  year  prior  to  the 
survey: 

•  Including  personnel  with  six  to  12  months  of  military 
service  and  asking  about  assaults  experienced  in  the 
prior  year  could  have  slightly  overestimated  military 
sexual  assaults  by  capturing  incidents  from  before 
these  personnel  entered  the  military.  However,  analyses 
suggest  that  this  had  a  very  small  effect  on  the  results, 
increasing  the  20,300  estimate  by  fewer  than  ten 
personnel. 

•  As  in  the  earlier  DoD  surveys  of  sexual  assault,  the 
RMWS  excluded  service  members  with  fewer  than  six 
months  of  service.  This  exclusion  could  have  led  to  an 
undercount  of  sexual  assault  incidents  in  the  prior  year 
by  25  to  190  personnel. 

•  Similarly,  the  RMWS  did  not  survey  those  who 
separated  from  the  military,  even  though  some  may 
have  experienced  a  sexual  assault  while  in  the  military 
in  the  year  prior  to  the  survey.  The  effect  of  this  source 
of  undercounting  is  potentially  the  largest.  Under 
reasonable  assumptions,  we  believe  this  source  of 
undercounting  could  have  caused  the  RMWS  estimates 
to  omit  900  to  2,800  service  members  with  sexual 
assaults  in  the  year  prior  to  the  survey. 

•  We  counted  as  sexual  assaults  both  those  who  had  an 
unwanted  sexual  experience  but  could  not  recall  the 
details  because  of  the  effects  of  alcohol  and  those  who 
stated  that  they  could  not  indicate  nonconsent  to  sexual 
activity  because  they  were  frozen  in  fear.  (We  received 
differing  expert  legal  opinions  on  whether  these  types  of 
incidents  constituted  sexual  assault  under  the  Uniform 
Code  of  Military  Justice.)  Excluding  all  such  cases 
would  reduce  the  RMWS-estimated  number  of  sexual 
assaults  by  about  50  individuals. 


By  far,  the  largest  source  of  potential  bias  was  the 
exclusion  of  service  members  who  had  separated  from 
the  military  before  the  survey  sample  was  drawn.  Thus, 
while  analyses  indicate  that  the  2014  RMWS  estimates 
accurately  reflected  the  sampled  population,  the  estimates 
were  more  likely  to  underestimate  than  overestimate  the 
true  count  of  sexual  assaults  among  all  who  served  in  the 
military  in  the  year  preceding  the  survey. 

Differences  in  events  counted  using  RMWS 
and  earlier  DoD  questions 

The  RAND  survey  form  and  the  prior  DoD  survey  form 
used  different  sets  of  questions  to  assess  sexual  assaults 
that  occurred  in  the  prior  year.  Both  sets  of  questions 
were  administered  in  2014  so  that  we  could  compare  the 
new  instrument  with  the  prior  instrument.  These  analyses 
showed  that  because  of  how  questions  were  framed  on 
the  earlier  form,  many  respondents  reported  incidents 
that  occurred  more  than  one  year  ago,  which  meant 
DoD’s  method  of  assessing  unwanted  sexual  contact 
likely  overestimated  its  prevalence  by  as  much  as 
20  percent.  The  DoD  assessment  method  also  identified 
only  about  half  as  many  service  members  who 
experienced  penetrative  assaults  as  the  RAND  form 
(4,200  versus  7,800).  This  effect  was  particularly  acute 
for  male  service  members  (1,200  versus  3,700),  likely 
because  the  RAND  form  identified  more  sexual  assaults 
that  occurred  in  the  context  of  hazing  or  that  were  not 
perceived  as  sexual  by  the  service  member.  The  RAND 
form,  which  did  not  require  respondents  to  know  the 
definition  of  sexual  harassment  and  correctly  apply  it 
to  their  experiences,  identified  far  more  cases  of  sexual 
harassment  than  the  form  previously  used  by  DoD. 


New  weighting  procedures  eliminated  more  potential  sources  of  bias  compared  with  previous  studies. 
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Sample  attrition,  breakoff,  and  tolerance 
of  the  survey  instrument 

Fewer  than  4  percent  of  those  who  began  the  RMWS 
survey  stopped  answering  questions,  or  “broke  off,” 
before  reaching  the  sexual  assault  module.  In  comparison, 
6.5  percent  of  those  who  received  the  prior  DoD  form 
as  part  of  the  2014  RAND  study  and  13.9  percent  of 
participants  in  the  DoD-administered  2012  survey  broke 
off  before  the  critical  questions.  Although  the  behaviorally 
and  anatomically  specific  language  in  the  RAND  form’s 
sexual  assault  module  offended  some  service  members, 
this  did  not  lead  to  a  higher  level  of  breakoffs  in  that 
section  relative  to  other  portions  of  the  survey.  The  RAND 
form  did  lead  to  more  complaints  than  the  prior  form, 
with  those  at  the  lowest  risk  for  sexual  assault  (men, 
officers,  and  those  in  more-senior  pay  grades)  tending 
to  object  to  the  survey  language  at  the  highest  rates.  We 
received  three  complaints  from  sexual  assault  victims 
that  the  survey  language  triggered  distressing  memories; 
all  said  they  discontinued  their  survey  participation.  In 
comparison,  12,210  service  members  classified  as  having 
experienced  a  sexual  assault  completed  the  survey.  As 
such,  we  concur  with  recent  findings  from  the  National 
Research  Council  that  the  benefits  of  using  behaviorally 
specific  language  to  more  accurately  assess  sexual  assault 
experiences  likely  outweigh  the  distress  associated 
with  exposing  survey  respondents  to  language  they  find 
offensive  or  disturbing. 
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